Empirical Methods in Information Extraction

Most corpus-basedmethods in natural language processing (NLP)were developed toprovide an arbitrary text-understanding application with one or more general-purpose linguistic capabilities. This is evident from the articles in this issue of AI Magazine. Charniak and Ng/Zelle, for example, describe techniques for part-of-speech tagging, parsing, and word-sense disambiguation. These techniques were created with no specific domain or high-level language-processing task in mind. In contrast, this article surveys the use of empirical methods for a particular natural language understanding task that is inherently domain-specific. The task is information extraction. Very generally, an information extraction system takes as input an unrestricted text and “summarizes” the text with respect to a prespecified topic or domain of interest: it finds useful information about the domain and encodes that information in a structured form, suitable for populating databases. In contrast to in-depth natural language understanding tasks, information extraction systems effectively skim a text to find relevant sections and then focus only on these sections in subsequent processing. The information extraction system in Figure 1, for example, summarizes stories about natural disasters, extracting for each such event the type of disaster, the date and time that it occurred, and data on any property damage or human injury caused by the event. Information extraction has figured prominently in the field of empirical NLP: The first largescale, head-to-head evaluations of NLP systems on the same text-understanding tasks were the DARPA-sponsored MUC performance evaluations of information extraction systems (Lehnert and Sundheim, 1991; Chinchor et al., 1993). Prior to each evaluation, all participating sites receive a corpus of texts from a predefined domain and the corresponding “answer keys” to use for system development. The answer keys are manually encoded templates — much like that of Figure 1 — that capture all information from the corresponding source text that is relevant to the domain, as specified in a set of written guidelines. After a short development phase, the NLP systems are evaluated by comparing the summaries each produces with the summaries generated by human experts for the same test set of previously unseen texts. The comparison is performed using an automated scoring program that rates each system according to measures of recall and precision. Recallmeasures the amount of the relevant information that theNLP system correctly extracts from the test collection while precision measures the reliability of the information extracted: